DataLiberation: add a streaming ShortcodeProcessor tokenizer - #300

Description

Introduces ShortcodeProcessor, a pull-based tokenizer and minimal editor for native WordPress shortcode markup.

Data Liberation needs to rewrite URLs without rendering content or changing unrelated bytes. Builder content can contain several nested languages:

[et_pb_section
    background_image="https://old.example/hero.jpg"
    custom_css_main_element='.x::before {
        content: "<";
        background: url(https://old.example/a.jpg) no-repeat center center fixed;
    }'
]
    [et_pb_text]<p>HTML and [contact-form-7 id="10"]</p>[/et_pb_text]
[/et_pb_section]

Treating the complete value as HTML can turn the CSS into content: "<";. Treating it as undifferentiated text cannot reliably separate a URL from adjacent CSS syntax. Running do_shortcode() is also unsuitable during an import: the original callbacks may be unavailable, and rendering discards the stored representation being migrated.

ShortcodeProcessor supplies the outer shortcode layer. It identifies shortcode and text regions, exposes shortcode attributes without normalizing them, and applies replacements to their original byte spans.

Architecture

The processor follows the restricted, forward-only model of WP_HTML_Tag_Processor:

1. next_token() advances through #shortcode and #text tokens.
2. next_shortcode() skips text and optionally filters by exact tag, tag prefix, closer policy, match offset, or escaped status.
3. A pull-based attribute iterator exposes each opener's original name, value, quote, offset, and length.
4. Mutations are queued as lexical replacements.
5. get_updated_text() applies those replacements while copying every unrelated byte unchanged.

It does not build a tree, invoke callbacks, or normalize the document. Opening and closing shortcodes are independent tokens, so same-name nesting remains visible:

$shortcodes = new ShortcodeProcessor(
	'[row level="1"][row level="2"]Inner[/row][/row]'
);

while ( $shortcodes->next_shortcode( 'row' ) ) {
	echo $shortcodes->is_tag_closer() ? '/row' : 'row';
}

The public API provides token location and text accessors; shortcode tag, closer, self-closing, and escaped state; attribute iteration, lookup, and replacement; and raw text-token replacement. It can update an existing attribute value or a complete text token, but does not add or remove attributes.

Where it applies

The class tokenizes a shortcode-bearing region. It does not guess whether an entire database field is HTML, CSS, JSON, serialized PHP, blocks, or shortcodes.

Divi's custom_css_* attributes illustrate the composition model:

$shortcodes = new ShortcodeProcessor( $post_content );

while (
	$shortcodes->next_shortcode(
		array(
			'tag_prefix'  => 'et_pb_',
			'tag_closers' => 'skip',
		)
	)
) {
	while ( $shortcodes->next_attribute() ) {
		$attribute_name = $shortcodes->get_attribute_name();
		if (
			null === $attribute_name ||
			0 !== strpos(
				strtolower( $attribute_name ),
				'custom_css_'
			)
		) {
			continue;
		}

		$css = new CSSURLProcessor(
			$shortcodes->get_attribute_value()
		);
		rewrite_css_urls( $css );
		$shortcodes->set_attribute_value(
			$css->get_updated_css()
		);
	}
}

This preserves content: "<", the closing ) in url(...), and declarations following the URL. Standalone Customizer or theme-option CSS should go directly to CSSURLProcessor; no shortcode pass is needed.

Parsing boundary

The processor finds shortcode candidates without WordPress or a registered shortcode table. Unlike Core's get_shortcode_regex(), it reports individual opener, closer, self-closing, and [[escaped]] tokens rather than matching a registered enclosing shortcode.

Quoted values may contain CSS, HTML, JSON, URLs, square brackets, and shortcode-like text. Named lookup is ASCII case-insensitive, positional attributes remain iterable, and the last duplicate named attribute wins. U+00A0 NO-BREAK SPACE and U+200B ZERO WIDTH SPACE are recognized as separators without changing the source.

[hidden] remains inherently ambiguous: it may be a shortcode, CSS selector, BBCode, or prose. Callers must isolate a shortcode-bearing region and should filter by registered names or known prefixes such as et_pb_, vc_, fusion_, or ct_.

Attribute updates preserve the existing delimiter when safe and switch delimiters when possible. If a new value requires quoting but contains both quote characters, the update returns false rather than emitting malformed markup.

Alternatives

• Core's shortcode regular expression starts from registered tags, matches enclosing content, and inherits documented nesting limitations. Imports need source spans without relying on the original runtime.
• do_shortcode() renders the content, runs plugin code, and destroys the builder representation.
• Whole-value HTML processing correctly escapes HTML text, but corrupts bytes that actually belong to CSS, JSON, or shortcode syntax.
• Builder-specific regular expressions repeat the same quote, bracket, nesting, and malformed-input problems while missing mapped third-party tags.

This class is not a sanitizer. Candidate matching does not prove that a tag is registered, and URL validation remains the URL-rewrite layer's responsibility.

Testing

The 35 focused tests and 94 assertions cover:

• forward-only traversal, UTF-8 byte offsets, all shortcode forms, and same-name nesting;
• queries, named and positional attributes, duplicate names, unusual whitespace, and quote switching;
• CSS, HTML, JSON, Base64, brackets, and shortcode-like text inside quoted values;
• malformed candidates, ambiguous CSS selectors, mixed blocks/HTML/CSS/shortcodes, and representative builder storage shapes;
• Divi CSS rewriting with content: "<", url(...) no-repeat, and percent-encoded UTF-8 paths.

The complete Data Liberation suite passes with 2,400 tests and 12,026 assertions. CI passes on PHP 7.2 through 8.5 across Linux, macOS, and Windows.

vendor/bin/phpunit components/DataLiberation/Tests/ShortcodeProcessorTest.php
vendor/bin/phpunit components/DataLiberation/Tests/
vendor/bin/phpcs components/DataLiberation/Shortcode/class-shortcodeprocessor.php

Follow-up

This PR supplies the tokenizer only. After it is merged and released, Reprint can classify fields from storage identity, plugin/post-meta signals, block markup, and known shortcode prefixes; decode structured containers; and dispatch each selected region to its owning processor.

Related: original WP_HTML_Tag_Processor proposal, CSSURLProcessor, and URLInTextProcessor punctuation handling.